Add Managed Kafka Connect Connectors Examples #13522

salmany · 2025-07-29T00:29:52Z

Note: This PR builds on top of the work in PR #13515 and should not be merged until that one is complete. The "Files changed" tab currently includes changes from the dependency.

This commit adds examples for creating Managed Kafka Connect connectors:

MirrorMaker connector
BigQuery sink connector
Cloud Storage sink connector
Pub/Sub sink connector
Pub/Sub source connector

Description

Fixes https://b.corp.google.com/issues/430087669

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

I have followed Sample Guidelines from AUTHORING_GUIDE.MD
README is updated to include all relevant information
Tests pass: nox -s py-3.9 (see Test Environment Setup)
Lint pass: nox -s lint (see Test Environment Setup)
These samples need a new API enabled in testing projects to pass (let us know which ones)
These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
Please merge this PR for me once it is approved

clusters * Adds code examples for creating, deleting, getting, listing and updating Managed Kafka Connect clusters

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

This commit adds examples for creating Managed Kafka Connect connectors: * MirrorMaker connector * BigQuery sink connector * Cloud Storage sink connector * Pub/Sub sink connector * Pub/Sub source connector

gemini-code-assist · 2025-07-29T00:30:06Z

Warning

Gemini encountered an error creating the summary. You can try again by commenting /gemini summary.

snippet-bot · 2025-07-29T00:51:26Z

Here is the summary of changes.

You are about to add 5 region tags.

managedkafka/snippets/connect/connectors/create_bigquery_sink_connector.py:58, tag managedkafka_create_bigquery_sink_connector
managedkafka/snippets/connect/connectors/create_cloud_storage_sink_connector.py:61, tag managedkafka_create_cloud_storage_sink_connector
managedkafka/snippets/connect/connectors/create_mirrormaker2_source_connector.py:60, tag managedkafka_create_mirrormaker2_source_connector
managedkafka/snippets/connect/connectors/create_pubsub_sink_connector.py:58, tag managedkafka_create_pubsub_sink_connector
managedkafka/snippets/connect/connectors/create_pubsub_source_connector.py:58, tag managedkafka_create_pubsub_source_connector

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

managedkafka/snippets/connect/connectors/create_mirrormaker_connector.py

managedkafka/snippets/connect/connectors/connectors_test.py

As per the [Authoring Guide] (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/AUTHORING_GUIDE.md#dependencies), each sample is required to have a requirements.txt file that lists the dependencies needed to run the sample.

Remove timeout to align with Managed Kafka Cluster update example: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/cdae4cacfe8f9612e554af11ef72bc8d34765ada/managedkafka/snippets/clusters/update_cluster.py#L60

glasnt · 2025-08-25T02:10:44Z

It looks like the linting here is failing because both the managedkafka/snippets and managedkafka/snippets/connect/connectors have requirements.txt files, which causes linting to run in both locations

Google linting style requires that imports from local files are grouped separately. But when the linting is running from a parent folder, it doesn't see those local files, so the lint from the parent folder fails due to child folder local imports.

It looks like #13515 also introduced a new requirements.txt file, so there's no longer just one requirements file for the whole snippets folder.

Since this PR adds a requirements.txt file that duplicates the parent requirements.txt, I'm going to remove it from this PR. This will mean:

changes to this folder will cause the whole managedkafka/snippets sample collection to be tested, which might cause future issues with changes causing unrelated tests to fire
the imports for the test file will conform to parent linting, but look potentially "out of order"

GoogleCloudPlatform#13522 (comment)

salmany · 2025-09-04T21:09:02Z

Re: the requirements.txt file, I had added it in the 'connectors' directory for running the tests locally, and was unsure if it was required to be submitted. Thanks for sorting it out!

managedkafka/snippets/connect/connectors/create_mirrormaker2_source_connector.py

rQ-Qrr · 2025-09-05T03:08:23Z

managedkafka/snippets/connect/connectors/create_pubsub_sink_connector.py

+    # connect_cluster_id = "my-connect-cluster"
+    # connector_id = "CPS_SINK_CONNECTOR_ID"
+    # topics = "GMK_TOPIC_ID"
+    # value_converter = "org.apache.kafka.connect.storage.StringConverter"


I think the default converter for CPS sink connector is org.apache.kafka.connect.converters.ByteArrayConverter on UI. Could you check it? Thanks!

This PR has already been merged. If you feel there are further changes needed, please create a new PR. Thanks!

Addressed in #13527 (comment)

rQ-Qrr · 2025-09-05T03:16:04Z

managedkafka/snippets/connect/connectors/create_mirrormaker2_source_connector.py

+        "target.cluster.alias": target_cluster_alias,  # This is usually the primary cluster.
+        # Replicate all topics from the source
+        "topics": topics,
+        # The value for bootstrap.servers is a comma-separated list of hostname:port pairs


I'm not quite sure if we could do that. I never tried to configure a comma-separated list for the source or target servers but only a single server address. Could you double check?

It would be better if adding an example of the server address format in the comment

Addressed in #13527 (comment)

salmany and others added 11 commits July 24, 2025 00:36

Add Managed Kafka Connect code samples for

c928344

clusters * Adds code examples for creating, deleting, getting, listing and updating Managed Kafka Connect clusters

Update google-cloud-managedkafka version to 0.1.12

8f2224a

Update managedkafka/snippets/connect/clusters/create_connect_cluster.py

1feb354

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update managedkafka/snippets/connect/clusters/create_connect_cluster.py

5840336

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update managedkafka/snippets/connect/clusters/delete_connect_cluster.py

d0df10f

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update managedkafka/snippets/connect/clusters/get_connect_cluster.py

505759a

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update managedkafka/snippets/connect/clusters/list_connect_clusters.py

d3fba2c

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update managedkafka/snippets/connect/clusters/update_connect_cluster.py

0a85a8b

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Add timeouts and improve error handling.

dfa07d2

Addressed PR comments.

ab697bf

Add Managed Kafka Connect Connectors examples

8af6e36

This commit adds examples for creating Managed Kafka Connect connectors: * MirrorMaker connector * BigQuery sink connector * Cloud Storage sink connector * Pub/Sub sink connector * Pub/Sub source connector

product-auto-label bot added the samples Issues that are directly related to samples. label Jul 29, 2025

salmany changed the title ~~Mkc-connectors-examples~~ Add Managed Kafka Connect Connectors Examples Jul 29, 2025

Fix import statements and add headers.

bc121bd

salmany marked this pull request as ready for review July 29, 2025 00:51

salmany requested review from a team as code owners July 29, 2025 00:51

salmany mentioned this pull request Jul 30, 2025

MKC Examples - Connector Operations #13527

Open

9 tasks

rQ-Qrr reviewed Aug 6, 2025

View reviewed changes

salmany and others added 8 commits August 7, 2025 22:25

Fix sample configs and update MM2 connector.

b9a15ef

Remove unwanted local test artifact.

a26f7b9

Adds requirements.txt file for samples.

af7a628

As per the [Authoring Guide] (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/AUTHORING_GUIDE.md#dependencies), each sample is required to have a requirements.txt file that lists the dependencies needed to run the sample.

Remove timeout in update_connect_cluster.py

1ae987f

Remove timeout to align with Managed Kafka Cluster update example: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/cdae4cacfe8f9612e554af11ef72bc8d34765ada/managedkafka/snippets/clusters/update_cluster.py#L60

Fix CPU and memory values for Cluster creation.

3e788a6

Fixed comment for minimum vCpu.

e908ecd

Fix lint by adding space.

1ede52b

Merge branch 'main' into mkc-connectors-examples

fac49a9

lint

3828949

glasnt added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Aug 25, 2025

kokoro-team removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Aug 25, 2025

lint

8fa4441

glasnt added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 25, 2025

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 25, 2025

lint

9271bc7

glasnt added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 25, 2025

glasnt self-assigned this Aug 25, 2025

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 25, 2025

resolve conflicting lint reports

230d0b5

GoogleCloudPlatform#13522 (comment)

glasnt added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 25, 2025

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 25, 2025

glasnt requested a review from rQ-Qrr August 25, 2025 02:23

glasnt added the waiting-response Waiting for the author's response. label Aug 25, 2025

Minor changes to MM2 sample and test.

882aa3d

glasnt added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed waiting-response Waiting for the author's response. labels Sep 4, 2025

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 4, 2025

glasnt reviewed Sep 4, 2025

View reviewed changes

managedkafka/snippets/connect/connectors/create_mirrormaker2_source_connector.py Outdated Show resolved Hide resolved

lint

697af57

glasnt added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 4, 2025

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 4, 2025

glasnt approved these changes Sep 4, 2025

View reviewed changes

glasnt merged commit 5426f61 into GoogleCloudPlatform:main Sep 4, 2025
11 checks passed

rQ-Qrr reviewed Sep 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Managed Kafka Connect Connectors Examples #13522

Add Managed Kafka Connect Connectors Examples #13522

Uh oh!

salmany commented Jul 29, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Jul 29, 2025

Uh oh!

snippet-bot bot commented Jul 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glasnt commented Aug 25, 2025

Uh oh!

salmany commented Sep 4, 2025

Uh oh!

Uh oh!

Uh oh!

rQ-Qrr Sep 5, 2025

Uh oh!

glasnt Sep 5, 2025

Uh oh!

salmany Sep 8, 2025

Uh oh!

rQ-Qrr Sep 5, 2025

Uh oh!

salmany Sep 8, 2025

Uh oh!

Uh oh!

Add Managed Kafka Connect Connectors Examples #13522

Add Managed Kafka Connect Connectors Examples #13522

Uh oh!

Conversation

salmany commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

gemini-code-assist bot commented Jul 29, 2025

Uh oh!

snippet-bot bot commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glasnt commented Aug 25, 2025

Uh oh!

salmany commented Sep 4, 2025

Uh oh!

Uh oh!

Uh oh!

rQ-Qrr Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

glasnt Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

salmany Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

rQ-Qrr Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

salmany Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

salmany commented Jul 29, 2025 •

edited

Loading

snippet-bot bot commented Jul 29, 2025 •

edited

Loading